Field of the Invention
The present invention relates to an image pickup apparatus which is capable of modeling three-dimensional data.
Description of the Related Art
It is desirable that three-dimensional modeling or space mapping is achieved by using image information or two-dimensional depth information that are acquired by a hand-held camera. In this case, with the use of optical vibration-proof control (image stabilization (IS) control), a motion blur and a motion distortion of an image caused by a hand shake can be reduced, and thus a deterioration of the three-dimensional model and the space map due to the motion blur and the motion distortion can be suppressed. The optical vibration-proof control is especially effective in a photographing scene such as an indoor scene in which a light amount is insufficient, a cloudy weather scene, and a night scene.
U.S. Pat. No. 8,471,915 discloses an image pickup system that corrects a rolling shutter distortion by using optical vibration-proof control and that suppresses a motion blur. Japanese Patent Laid-open No. 2014-110043 discloses an apparatus that generates a three-dimensional model by combining a plurality of partial three-dimensional models when it is necessary to capture a large object that cannot be contained in a field of view or take a close-up photo. Japanese Translation of PCT International Application Publication No. 2014-511591 discloses a method of measuring or calculating position or attitude information of a camera or change information based on a certain origin at the moment of capturing an image or depth information to be used for integration processing. A method of calculating the position or the attitude of the camera during the photography by using a captured image or information acquired by an attitude sensor provided in the camera is called a self-position estimation.
However, when three-dimensional modeling is performed by integrating a color image with an image including depth distance information as disclosed in Japanese Patent Laid-open No. 2014-110043 and Japanese Translation of PCT International Application Publication No. 2014-511591, a displacement occurs even if RGB data captured from each viewpoint are reversely projected in a unified coordinate space to be merged on condition that the optical vibration-proof control is ON. This is because perspective that occurs by a perspective projection and that depends on an optical vibration-proof state is geometrically erroneous. Since a position change (attitude change) of a camera cannot be correctly estimated from an image, it is difficult to improve a quality of the three-dimensional modeling while performing the optical vibration-proof control.